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Emotions play important part in mathematical problem solving, yet the theories of 
their role are still at their preliminary stages. In our study, we introduce a method, 
where screen recordings and automatic emotion recognition software are used to 
study the emotional states of five upper secondary school students during a solitary 
GeoGebra problem solving session. Common emotional states during problem solving 
were neutral (40 % of time), sad (34 % of time), happy (15 % of time) and angry (8 % 
of time). Different phases of problem solving were emotionally different, non-neutral 
emotional states being most prevalent during decision points such as using the undo 
button. The method used opens possibilities for new kinds of research designs for 
studying the role of emotions in problem solving. 


INTRODUCTION 


Affective elements have received much attention in the literature of mathematics 
education in general and problem solving in particular (e.g. McLeod & Adams 1989, 
Schoenfeld 1985, Leder, Pehkonen & Torner 2002, DeBellis & Goldin 2006). 
However, the majority of studies have focused on relatively stable traits in the affective 
domain, such as attitudes, beliefs and values. Considerably less attention has been 
given to emotions, defined as “rapidly changing states of feeling experienced 
consciously or occurring preconsciously or unconsciously during mathematical (or 
other) activity” (DeBellis & Goldin 2006, p. 135). 


Emotions are influential in the key moments that determine the success of solving a 
problem. Goldin (2000) uses the concept of affective pathways to describe how the 
typical patterns of emotional states lead to successful or unsuccessful problem solving 
behavior, and in the long run partly shape one's attitudes, beliefs and values concerning 
mathematics. However, proper understanding about the role of emotional states 
requires further research. 


Studying momentary emotional states is typically work-intensive and therefore the 
number of subjects in such studies is often small. In this report, we introduce a method 
that automatizes part of the work and thus opens possibilities for new kinds of research 
designs: we use screen recording technology to capture student’s computer-aided 
problem solving process and automatic emotion recognition software to analyze 
student’s emotions during the process. 
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Phases of problem solving 


For research purposes it is often useful to identify different phases of the problem 
solving process. Several models of phases or stages of problem solving have been 
introduced by different authors (e.g. Polya 1945; Mason, Burton & Stacey 1982; 
Schoenfeld 1985; Hahkioniemi, Leppaaho & Francisco 2013). Polya's (1945) model 
with four phases — understanding the problem, devising a plan, carrying out the plan 
and looking back — 1s most widely recognized, but it is intended to be rather a guide to 
a problem solver than a research tool. Although the same applies to Mason's et al. 
(1982) model, they importantly point out that problem solving process doesn't 
necessarily proceed linearly along the phases but the solver might for example return to 
make sense of the problem after some new information has occurred to him or her. 


In his studies, Schoenfeld (1985) used transcriptions of students' discussions during 
problem solving to identify the phases of problem solving. He distinguishes six phases 
in problem solving: reading, analyzing, exploring, planning, implementing, and 
verifying. Essential to his model are also decision points — moments in the problem 
solving process when a student should use metacognitive skills to decide on further 
actions. For example, when new information concerning the problem occurs to the 
student, he or she must consider whether the current attempt to solve the problem is 
still valid or should a new strategy be used instead. 


The two main approaches to collect data on student thinking during the process of 
problem solving have been think-aloud and stimulated recall methods. However, as 
Hahkioniemi, Leppaaho and Francisco (2013) indicate, it is possible to study problem 
solving with computer software using only screen recordings as the base for analysis. 
Hahkioniemi et al. (2013) investigated problem solving with dynamic geometry 
software GeoGebra. They found none of the previous models of problem solving 
directly applicable, and thus developed a new classification, consisting of five phases: 
framing the problem, exploring the solution, conjecturing, investigating the 
conjecture, and justifying. There was great variation between students on how they 
moved through these phases: some students proceeded linearly from phase to phase, 
but most of them skipped phases and/or returned to previous phases during the process. 


In present study, an adaptation of Schoenfeld's (1985) model is used. Instead of 
discussion or think-aloud transcriptions, screen recordings are used to conduct the 
analysis, similarly to Hahki6niem1’s et al. (2013) study. 

Emotions and problem solving 

There is a general agreement that emotions have an important role in human learning 
and mathematical problem solving specifically (Hannula 2012). However, theories 


about the role of emotions in the process of problem solving are still at their 
preliminary stages (Lehman et. al. 2008, Goldin, Epstein, Schorr & Warner 2011). 


It is well established that emotions direct attention and bias cognitive processing. For 
example, fear (anxiety) directs attention towards threatening information and sadness 
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(depression) biases memory towards a less optimistic view of the past (Power and 
Dalgleish 1997). There is also indication that positive emotions promote the creative 
aspects of problem solving, while negative emotions facilitate the reliable memory 
retrieval and performance of routines (Pekrun & Stephens, 2010). In mathematical 
problem solving, curiosity, puzzlement, bewilderment, frustration, pleasure, elation, 
satisfaction, anxiety, and despair have been observed to have important self-regulative 
functions (DeBellis & Goldin 2006). 


There are several theories concerning emotion, emerging from different research 
traditions. In this paper, we follow the Darwinian tradition, where emotions are seen as 
products of evolution, they can be categorized to a small number of universal basic 
emotions (anger, disgust, fear, happiness, sadness and surprise) and these emotions can 
be identified from facial expressions (Ekman & Friesen 1971; Ekman 1992). This 
approach allows us to use Facial Action Coding System (FACS; Ekman & Friesen 
1978) to identify emotions from students’ facial expressions. The process of 
identifying emotions has traditionally required trained human coders, but during last 
few decades, automated computer methods have been developed (Bettadapura 2012). 


There is some evidence that basic emotions would be rare in learning context. (Craig, 
D’Mello, Witherspoon & Graesser 2008; Lehman, D’Mello & Person 2008). 
Therefore, alternative emotion classifications that would suit better the learning setting 
have been developed. Both Craig et al. (2008) and Lehman et al. (2008) used a 
learning-centered emotion classification consisting of anxiety, boredom, confusion, 
contempt, curiosity, eureka and frustration. Pekrun and Stephens (2010) describe a 
model for emotion in achievement setting, which includes enjoyment, relaxation, 
anger, frustration, boredom, hope, joy, relief, anxiety, hopelessness, pride, gratitude, 
contentment, shame, sadness and disappointment. 


Hannula (2012) suggests that achievement emotions can also be looked through six 
basic emotions. Enjoyment, hope, joy, pride, and gratitude are different variants of 
happiness, whereas boredom, hopelessness, sadness, and disappointment are variants 
of sadness. Frustration is a variant of anger and anxiety a variant of fear. Relaxation, 
relief and contentment do not present any basic emotion, but they can be seen as 
removal of a negative emotion. (Hannula 2012). 


In this study, our aim is to find out, how different basic emotions occur in different 
phases of problem solving. Another important aim is to investigate, how our research 
methodology, which combines screen recording and automatic emotion recognition, 
works in the context of mathematical problem solving. 


METHODS 


Data collection 


The data for this study was collected in a Finnish upper secondary school in Helsinki. 
Participants were five students (two girls, three boys) participating in the advanced 
mathematics syllabus. Each student separately participated in a session lasting about 
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one hour, of which 20 minutes was devoted to problem solving. The session consisted 
of getting acquainted with the dynamic geometry software GeoGebra (Hohenwarter, 
2002) using a practice applet, working with a two-part mathematical problem using 
two GeoGebra applets, and discussing the process with the researcher. 


The mathematical background of the problem was the circumscribed circle of a 
triangle. First part of the problem was to find out, whether a circle could be drawn so 
that it goes through three points, in three given situations. The following instructions 
were given: 
Let A=(-2,0), B=(0,2), C=(2,0), D=(0,0) and E=(3,-2). Is it possible to draw a circle so that 
the circle goes through a) A, B and C, b) B, C or D, c) C, D and E? 


The GeoGebra applet provided with the problem included the points mentioned in the 
instructions, a coordinate system and a customized toolbar with following tools: Move, 
Delete, Point, Perpendicular Line, Perpendicular Bisector, Angle Bisector, Polygon 
and Circle with Center through Point. 


In the second part of the problem, the GeoGebra applet contained the same toolbar but 
no coordinate system. Students were given a statement "It is always possible to draw a 
circle through three given points" and asked to either show that it 1s true or show that it 
is false. 


Each student's screen was recorded during the use of GeoGebra applets. Integrated 
webcam of the laptop computer running GeoGebra was used to record a video of 
student's face. After the problem solving session, a video-based stimulated recall 
interview (e.g. DeBellis & Goldin 2006), was conducted and audio-recorded, but that 
data is not analyzed in this report. 


Analyzing the screen recordings 


We analyzed only the screen recordings of the two problems. Events on the screen 
were transcribed to a table with time codes. Transcriptions were read multiple times 
and similar events were grouped together. We ended up with the following event 
classes: changes to next or previous applet, chooses a new tool, clicks the undo button, 
deletes an object, draws an object, explores the menu, holds the pointer still, moves a 
point and moves the cursor. 


To identify the phases of problem solving, the screen recordings were watched again, 
looking for phases described in the literature. By making interpretations of students' 
actions (Table 1), events representing Schoenfeld's (1985) characterizations of 
reading, analyzing, exploring, implementing and verifying could be found in the data. 
Moments when a student used GeoGebra's delete tool or the undo button, or changed to 
the next or the previous applet, were considered decision points. Each event in the 
transcription was encoded to belong to one of the phases or to be a decision point. 
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Student action Phase 
Student holds the pointer still above instructions Read 
Student holds the pointer still or moves it around without concentrating on Analyze 
any particular object 
Student draws objects, moves points or objects or explores the menu Explore 
Student performs a series of actions Implement 
After accomplishing a goal, student holds the pointer still or redoes the Verify 
actions that lead to success 
Student uses the delete tool, the undo button, or changes to next or previous Decision 
applet point 


Table 1: Interpretation rules for student actions. 
Automatic emotion recognition 


Noldus FaceReader 5 was used to recognize emotions from student's face videos. The 
software is based on the theory of basic emotions and it has been used e.g. in studies of 
usability (Goldberg 2012), intelligent tutoring systems (Harley, Bouchet & Azevedo 
2012) and consumer behaviour (He, Boesveldt, Graaf, & Wijk 2012). FaceReader 
identifies key points in subject's face and classifies the emotions using an artificial 
neural network trained with manually notated images (Loiyens & Krips 2014). In 
addition to six basic emotions, the classification includes an emotionally neutral state. 
As an output, FaceReader produces the intensity of each emotional state 10-30 times 
per second and the dominating emotional state at each time. 


In our analysis, for each face video we first used FaceReader's automatic calibration 
and then ran the analysis. FaceReader state logs were used to add emotional data to the 
event transcription. Whenever multiple emotional states occurred during a single 
event, all the states were recorded (Table 2). 


Time code Event Emotional state 
15:34 Student draws an angle bisector at the intersect of the Sad 
perpendicular bisectors of AD and BD 
15:38 Student clicks the undo button Sad/Angry 
15:45 Student explores the menu Angry/Neutral 


Table 2: GeoGebra event transcription with emotional states. 


All event-emotion combinations were extracted from the transcriptions and separately 
cross tabulated with the classifications of the event classes and phases of problem 
solving. A chi-squared test was calculated for both contingency tables. 
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RESULTS 


Most common emotional states during problem solving were neutral (40 % of time) 
and sad (34 % of time). Happy (15 % of time) and angry (8 % of time) were also 
common, whereas surprised (3 % of time), disgusted (1 % of time) and scared (0% of 
time) were rare. 


Cross tabulation of event classes and emotional states is presented in Table 3. 
Surprised, disgusted and scared with 12, 7 and 0 event-emotion combinations 
respectively were so rare that they were excluded from further analysis. Small 
expected frequency prevented using a chi-squared test. 


Event-emotion combinations Neutral Happy Sad Angry Total 
Changes to next or previous applet 3 1 5 1 10 
Chooses a new tool Pie, 16 30 7 78 
Clicks the undo button 0 > 10 10 2 
Deletes an object 3 2 3 3 11 
Draws an object 24 17 29 7 7 
Holds the pointer still 27 11 19 3 60 
Moves a point 19 11 17 1 48 
Moves the cursor 21 9 19 4 53 
Total 126 BB 139 4] 379 


Table 3: Contingency table of the event classes and the emotional states 


The cross tabulation of the phases of problem solving and the emotional states is 
presented in Table 4. Different phases were emotionally different beyond coincidence 
(chi-squared test p=0.004). 


-Event-emotion combinations Neutral Happy Sad ~—s Angry _—‘ Total _ 
Read 1] 7 12 2 32 
Analyze 21 3 10 3 37 
Explore 58 42 70 20 190 
Implement 19 1 20 4 44 
Verify 10 6 Ti 2 29 
Decision points 7 14 16 10 47 
Total 126 fk 139 4] 379 


Table 4: Contingency table of the phases of problem solving and the emotional states 
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DISCUSSION 


Our results suggest, that although neutral was the most common emotional state in this 
study, also some of the basic emotions (happiness, sadness and anger) are common in 
problem solving. If achievement emotions, described by Pekrun and Stephens (2010), 
are looked through basic emotions as Hannula (2012) suggests, most of the 
achievement emotions are covered by these three emotional states. 


Results also indicate that different events and the phases of problem solving are 
emotionally different. However, more research is needed to investigate these 
differences in detail. The method for classifying phases of problem solving used in this 
study, although being straightforward, includes multiple presumptions about student 
thinking. 


Goldin (2000) suggests that emotions are in an important role during the key moments 
of problem solving process. This is in line with our findings about the emotional states 
associated with decision points (Schoenfeld 1985): the proportion of non-neutral 
emotional states (happiness, sadness and anger) was greatest during the decision points 
(Table 4). 


CONCLUSIONS 


The aim of this study was to use a combination of screen recording analysis and 
automatic emotion recognition to analyze students' emotions in different phases of 
problem solving. Our results, indicating that different phases are emotionally different, 
are encouraging and suggest that this kind of methodology can be used to study 
mathematical problem solving. More studies with a larger number of subjects are 
needed to further investigate the potential of this research design. 
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